<!DOCTYPE html><html><head><title>README</title><link href="http://www.jboss.org/jdf/stylesheets/documentation.css" rel="stylesheet"></link><link href="http://www.jboss.org/jdf/stylesheets/pygments.css" rel="stylesheet"></link></head><body>
<h1><a id="jtsdistributedcrashrec-jts-and-distributed-crash-recovery-" class="anchor" href="#jtsdistributedcrashrec-jts-and-distributed-crash-recovery-"><span class="anchor-icon"></span></a>jts-distributed-crash-rec: JTS and distributed crash recovery </h1>

<p>Author: Tom Jenkinson<br/>
Level: Advanced<br/>
Technologies: JTS<br/>
Summary: Demonstrates recovery of distributed crashed components<br/>
Prerequisites: jts<br/>
Target Product: EAP<br/>
Product Versions: EAP 6.1, EAP 6.2<br/>
Source: <a href="https://github.com/jboss-developer/jboss-eap-quickstarts/">https://github.com/jboss-developer/jboss-eap-quickstarts/</a>  </p>

<h2><a id="what-is-it" class="anchor" href="#what-is-it"><span class="anchor-icon"></span></a>What is it?</h2>

<p>This quickstart demonstrates a distributed crash recovery across multiple application servers. </p>

<p>Crash recovery is a key feature provided by an application server and ensures an application&rsquo;s data consistency, even in the presence of failure. Providing reliable crash recovery helps qualify the pedigree of an application server, distributed crash recovery even more so.</p>

<p>This quickstart uses the application components from the <code>jts</code> quickstart. It provides a Byteman rule to inject a halt into the application server at a crucial point in the phase2 commit of the transaction. Byteman is used solely to raise an artificial fault. An IDE could also provide this simulation, although it is more complex to use an IDE for this purpose.</p>

<p>Apart from that, this quickstart works the same as the <code>jts</code> quickstart and if the Byteman rule is left out, this quickstart <em>is</em> the JTS quickstart.</p>

<p>As an overview, the sequence of events to expect:</p>

<ol>
<li>Configure and start two JBoss servers.</li>
<li>Build and deploy the two application components.</li>
<li>Open a web browser and attempt to invoice two customers as with the <code>jts</code> quickstart.</li>
<li>JBoss server 1 will run through a two-phase commit (2PC), preparing the resources in JBoss server 1 and JBoss server 2. JBoss server 1 will then crash before it can call commit.</li>
<li>The user is invited to inspect the content of the transaction objectstore.</li>
<li>JBoss server 1 should be restarted. It will then recover the &ldquo;invoices&rdquo; delivered to the MDBs, just as it does in the <em>jts</em> quickstart</li>
</ol>

<h2><a id="system-requirements" class="anchor" href="#system-requirements"><span class="anchor-icon"></span></a>System requirements</h2>

<p>The application this project produces is designed to be run on Red Hat JBoss Enterprise Application Platform 6.1 or later. </p>

<p>All you need to build this project is Java 6.0 (Java SDK 1.6) or later, Maven 3.0 or later.</p>

<h2><a id="configure-maven" class="anchor" href="#configure-maven"><span class="anchor-icon"></span></a>Configure Maven</h2>

<p>If you have not yet done so, you must <a href="../README.html#configure-maven">Configure Maven</a> before testing the quickstarts.</p>

<h2><a id="configure-byteman" class="anchor" href="#configure-byteman"><span class="anchor-icon"></span></a>Configure Byteman</h2>

<p>This quickstart uses Byteman to help demonstrate crash recovery. Instructions to install and configure Byteman can be found here: <a href="../README.html#install-and-configure-byteman">Install and Configure Byteman</a></p>

<h2><a id="prerequisites" class="anchor" href="#prerequisites"><span class="anchor-icon"></span></a>Prerequisites</h2>

<p>Developers should be familiar with the concepts introduced in the following quickstarts:</p>

<ul>
<li><em>cmt</em></li>
<li><em>jts</em></li>
</ul>

<p>IMPORTANT: This quickstart depends on the deployment of the <code>jts</code> quickstart for its test. Before running this quickstart, see the <a href="../jts/README.html">jts README</a> file for details on how to deploy it.</p>

<p>You can verify the deployment of the <code>jts</code> quickstart by accessing the following URL:  <a href="http://localhost:8080/jboss-jts-application-component-1/">http://localhost:8080/jboss-jts-application-component-1/</a>.</p>

<h2><a id="clear-the-transaction-objectstore" class="anchor" href="#clear-the-transaction-objectstore"><span class="anchor-icon"></span></a>Clear the Transaction ObjectStore</h2>

<p>Make sure there is no transaction objectstore data left after testing this or any of the other quickstarts. If you are using the default file based transaction logging store:</p>

<ol>
<li><p>Open a command line and type the following:</p>
<div class="highlight"><pre><span class="n">ls</span> <span class="err">$</span><span class="n">JBOSS_HOME</span><span class="o">/</span><span class="n">standalone</span><span class="o">/</span><span class="n">data</span><span class="o">/</span><span class="n">tx</span><span class="o">-</span><span class="n">object</span><span class="o">-</span><span class="n">store</span><span class="o">/</span><span class="n">ShadowNoFileLockStore</span><span class="o">/</span><span class="n">defaultStore</span><span class="o">/</span><span class="n">StateManager</span><span class="o">/</span><span class="n">BasicAction</span><span class="o">/</span><span class="n">TwoPhaseCoordinator</span><span class="o">/</span><span class="n">AtomicAction</span><span class="o">/</span>
</pre></div></li>
<li><p>If this directory exists and contains any files, delete them before starting the server:</p>
<div class="highlight"><pre><span class="n">rm</span> <span class="o">-</span><span class="n">rf</span> <span class="err">$</span><span class="n">JBOSS_HOME</span><span class="o">/</span><span class="n">standalone</span><span class="o">/</span><span class="n">data</span><span class="o">/</span><span class="n">tx</span><span class="o">-</span><span class="n">object</span><span class="o">-</span><span class="n">store</span><span class="o">/</span><span class="n">ShadowNoFileLockStore</span><span class="o">/</span><span class="n">defaultStore</span><span class="o">/</span><span class="n">StateManager</span><span class="o">/</span><span class="n">BasicAction</span><span class="o">/</span><span class="n">TwoPhaseCoordinator</span><span class="o">/</span><span class="n">AtomicAction</span><span class="o">/*</span>
</pre></div></li>
<li><p>On Windows, use the file manager to accomplish the same result.</p></li>
</ol>

<h2><a id="test-the-application" class="anchor" href="#test-the-application"><span class="anchor-icon"></span></a>Test the Application</h2>

<ol>
<li><p>If you have not yet done so, configure the two application servers and deploy the <code>jts</code> quickstart. Follow the instructions in the <a href="../jts/README.html">jts README</a> file.</p></li>
<li><p>Configure Byteman to halt JBoss server 1</p>

<ul>
<li>Stop the JBoss servers.</li>
<li><a href="#clear-the-transaction-objectstore">Clear the transaction objectstore data</a> remaining from previous tests.</li>
<li><p>Follow the instructions to <a href="../README.html#halt-the-application-using-byteman">halt application server 1 using Byteman</a>. The following 2 lines of text will be appended to the server configuration file for server 1 only:</p>
<div class="highlight"><pre>For Linux: 
      JAVA_OPTS=&quot;-javaagent:/PATH_TO_BYTEMAN_DOWNLOAD/lib/byteman.jar=script:/PATH_TO_QUICKSTARTS/jts-distributed-crash-rec/byteman-scripts/failAfterPrepare.btm <span class="cp">${</span><span class="n">JAVA_OPTS</span><span class="cp">}</span>&quot;
      JAVA_OPTS=&quot;-Dorg.jboss.byteman.transform.all -Djboss.modules.system.pkgs=org.jboss.byteman -Dorg.jboss.byteman.verbose=true <span class="cp">${</span><span class="n">JAVA_OPTS</span><span class="cp">}</span>&quot;

For Windows: 
      JAVA_OPTS=%JAVA_OPTS% -javaagent:C:PATH_TO_BYTEMAN_DOWNLOAD\lib\byteman.jar=script:C:\PATH_TO_QUICKSTARTS\jts-distributed-crash-rec\byteman-scripts\failAfterPrepare.btm %JAVA_OPTS%
      JAVA_OPTS=%JAVA_OPTS% -Dorg.jboss.byteman.transform.all -Djboss.modules.system.pkgs=org.jboss.byteman -Dorg.jboss.byteman.verbose=true 
</pre></div></li>
</ul></li>
<li><p>Start both of the JBoss servers</p>

<p>If you are using Linux:</p>
<div class="highlight"><pre><span class="n">Server</span> <span class="mi">1</span><span class="o">:</span> <span class="n">JBOSS_HOME_SERVER_1</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">standalone</span><span class="p">.</span><span class="n">sh</span> <span class="o">-</span><span class="n">c</span> <span class="n">standalone</span><span class="o">-</span><span class="n">full</span><span class="p">.</span><span class="n">xml</span>
<span class="n">Server</span> <span class="mi">2</span><span class="o">:</span> <span class="n">JBOSS_HOME_SERVER_2</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">standalone</span><span class="p">.</span><span class="n">sh</span> <span class="o">-</span><span class="n">c</span> <span class="n">standalone</span><span class="o">-</span><span class="n">full</span><span class="p">.</span><span class="n">xml</span> <span class="o">-</span><span class="n">Djboss</span><span class="p">.</span><span class="n">socket</span><span class="p">.</span><span class="n">binding</span><span class="p">.</span><span class="n">port</span><span class="o">-</span><span class="n">offset</span><span class="o">=</span><span class="mi">100</span>
</pre></div>
<p>If you are using Windows</p>
<div class="highlight"><pre><span class="n">Server</span> <span class="mi">1</span><span class="o">:</span> <span class="n">JBOSS_HOME_SERVER_1</span><span class="err">\</span><span class="n">bin</span><span class="err">\</span><span class="n">standalone</span><span class="p">.</span><span class="n">bat</span> <span class="o">-</span><span class="n">c</span> <span class="n">standalone</span><span class="o">-</span><span class="n">full</span><span class="p">.</span><span class="n">xml</span>
<span class="n">Server</span> <span class="mi">2</span><span class="o">:</span> <span class="n">JBOSS_HOME_SERVER_2</span><span class="err">\</span><span class="n">bin</span><span class="err">\</span><span class="n">standalone</span><span class="p">.</span><span class="n">bat</span> <span class="o">-</span><span class="n">c</span> <span class="n">standalone</span><span class="o">-</span><span class="n">full</span><span class="p">.</span><span class="n">xml</span> <span class="o">-</span><span class="n">Djboss</span><span class="p">.</span><span class="n">socket</span><span class="p">.</span><span class="n">binding</span><span class="p">.</span><span class="n">port</span><span class="o">-</span><span class="n">offset</span><span class="o">=</span><span class="mi">100</span>
</pre></div></li>
<li><p>Access the application at the following URL: <a href="http://localhost:8080/jboss-jts-application-component-1/">http://localhost:8080/jboss-jts-application-component-1/</a></p>

<ul>
<li><p>When you enter a name and click to &ldquo;add&rdquo; that customer, you will see the following in the application server 1 console:</p>
<div class="highlight"><pre><span class="mi">15</span><span class="o">:</span><span class="mi">46</span><span class="o">:</span><span class="mi">55</span><span class="o">,</span><span class="mi">070</span> <span class="n">INFO</span>  <span class="o">[</span><span class="n">org</span><span class="o">.</span><span class="na">jboss</span><span class="o">.</span><span class="na">ejb</span><span class="o">.</span><span class="na">client</span><span class="o">]</span> <span class="o">(</span><span class="n">http</span><span class="o">-</span><span class="n">localhost</span><span class="o">-</span><span class="mf">127.0</span><span class="o">.</span><span class="mf">0.1</span><span class="o">-</span><span class="mi">8080</span><span class="o">-</span><span class="mi">1</span><span class="o">)</span> <span class="n">JBoss</span> <span class="n">EJB</span> <span class="n">Client</span> <span class="n">version</span> <span class="mf">1.0</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="na">Beta12</span>
<span class="mi">15</span><span class="o">:</span><span class="mi">46</span><span class="o">:</span><span class="mi">55</span><span class="o">,</span><span class="mi">658</span> <span class="n">INFO</span>  <span class="o">[</span><span class="n">stdout</span><span class="o">]</span> <span class="o">(</span><span class="n">http</span><span class="o">-</span><span class="n">localhost</span><span class="o">-</span><span class="mf">127.0</span><span class="o">.</span><span class="mf">0.1</span><span class="o">-</span><span class="mi">8080</span><span class="o">-</span><span class="mi">1</span><span class="o">)</span> <span class="n">Rule</span><span class="o">.</span><span class="na">execute</span> <span class="n">called</span> <span class="k">for</span> <span class="n">Fail</span> <span class="mi">2</span><span class="n">PC</span> <span class="n">after</span> <span class="n">prepare_0</span>
<span class="mi">15</span><span class="o">:</span><span class="mi">46</span><span class="o">:</span><span class="mi">55</span><span class="o">,</span><span class="mi">665</span> <span class="n">INFO</span>  <span class="o">[</span><span class="n">stdout</span><span class="o">]</span> <span class="o">(</span><span class="n">http</span><span class="o">-</span><span class="n">localhost</span><span class="o">-</span><span class="mf">127.0</span><span class="o">.</span><span class="mf">0.1</span><span class="o">-</span><span class="mi">8080</span><span class="o">-</span><span class="mi">1</span><span class="o">)</span> <span class="n">HelperManager</span><span class="o">.</span><span class="na">install</span> <span class="k">for</span> <span class="n">helper</span> <span class="n">classorg</span><span class="o">.</span><span class="na">jboss</span><span class="o">.</span><span class="na">byteman</span><span class="o">.</span><span class="na">rule</span><span class="o">.</span><span class="na">helper</span><span class="o">.</span><span class="na">Helper</span>
<span class="mi">15</span><span class="o">:</span><span class="mi">46</span><span class="o">:</span><span class="mi">55</span><span class="o">,</span><span class="mi">666</span> <span class="n">INFO</span>  <span class="o">[</span><span class="n">stdout</span><span class="o">]</span> <span class="o">(</span><span class="n">http</span><span class="o">-</span><span class="n">localhost</span><span class="o">-</span><span class="mf">127.0</span><span class="o">.</span><span class="mf">0.1</span><span class="o">-</span><span class="mi">8080</span><span class="o">-</span><span class="mi">1</span><span class="o">)</span> <span class="n">calling</span> <span class="n">activated</span><span class="o">()</span> <span class="k">for</span> <span class="n">helper</span> <span class="n">classorg</span><span class="o">.</span><span class="na">jboss</span><span class="o">.</span><span class="na">byteman</span><span class="o">.</span><span class="na">rule</span><span class="o">.</span><span class="na">helper</span><span class="o">.</span><span class="na">Helper</span>
<span class="mi">15</span><span class="o">:</span><span class="mi">46</span><span class="o">:</span><span class="mi">55</span><span class="o">,</span><span class="mi">666</span> <span class="n">INFO</span>  <span class="o">[</span><span class="n">stdout</span><span class="o">]</span> <span class="o">(</span><span class="n">http</span><span class="o">-</span><span class="n">localhost</span><span class="o">-</span><span class="mf">127.0</span><span class="o">.</span><span class="mf">0.1</span><span class="o">-</span><span class="mi">8080</span><span class="o">-</span><span class="mi">1</span><span class="o">)</span> <span class="n">Default</span> <span class="n">helper</span> <span class="n">activated</span>
<span class="mi">15</span><span class="o">:</span><span class="mi">46</span><span class="o">:</span><span class="mi">55</span><span class="o">,</span><span class="mi">667</span> <span class="n">INFO</span>  <span class="o">[</span><span class="n">stdout</span><span class="o">]</span> <span class="o">(</span><span class="n">http</span><span class="o">-</span><span class="n">localhost</span><span class="o">-</span><span class="mf">127.0</span><span class="o">.</span><span class="mf">0.1</span><span class="o">-</span><span class="mi">8080</span><span class="o">-</span><span class="mi">1</span><span class="o">)</span> <span class="n">calling</span> <span class="n">installed</span><span class="o">(</span><span class="n">Fail</span> <span class="mi">2</span><span class="n">PC</span> <span class="n">after</span> <span class="n">prepare</span><span class="o">)</span> <span class="k">for</span> <span class="n">helper</span> <span class="n">classorg</span><span class="o">.</span><span class="na">jboss</span><span class="o">.</span><span class="na">byteman</span><span class="o">.</span><span class="na">rule</span><span class="o">.</span><span class="na">helper</span><span class="o">.</span><span class="na">Helper</span>
<span class="mi">15</span><span class="o">:</span><span class="mi">46</span><span class="o">:</span><span class="mi">55</span><span class="o">,</span><span class="mi">667</span> <span class="n">INFO</span>  <span class="o">[</span><span class="n">stdout</span><span class="o">]</span> <span class="o">(</span><span class="n">http</span><span class="o">-</span><span class="n">localhost</span><span class="o">-</span><span class="mf">127.0</span><span class="o">.</span><span class="mf">0.1</span><span class="o">-</span><span class="mi">8080</span><span class="o">-</span><span class="mi">1</span><span class="o">)</span> <span class="n">Installed</span> <span class="n">rule</span> <span class="n">using</span> <span class="k">default</span> <span class="n">helper</span> <span class="o">:</span> <span class="n">Fail</span> <span class="mi">2</span><span class="n">PC</span> <span class="n">after</span> <span class="n">prepare</span>
<span class="mi">15</span><span class="o">:</span><span class="mi">46</span><span class="o">:</span><span class="mi">55</span><span class="o">,</span><span class="mi">668</span> <span class="n">INFO</span>  <span class="o">[</span><span class="n">stdout</span><span class="o">]</span> <span class="o">(</span><span class="n">http</span><span class="o">-</span><span class="n">localhost</span><span class="o">-</span><span class="mf">127.0</span><span class="o">.</span><span class="mf">0.1</span><span class="o">-</span><span class="mi">8080</span><span class="o">-</span><span class="mi">1</span><span class="o">)</span> <span class="n">Fail</span> <span class="mi">2</span><span class="n">PC</span> <span class="n">after</span> <span class="n">prepare</span> <span class="n">execute</span>
<span class="mi">15</span><span class="o">:</span><span class="mi">46</span><span class="o">:</span><span class="mi">55</span><span class="o">,</span><span class="mi">669</span> <span class="n">INFO</span>  <span class="o">[</span><span class="n">stdout</span><span class="o">]</span> <span class="o">(</span><span class="n">http</span><span class="o">-</span><span class="n">localhost</span><span class="o">-</span><span class="mf">127.0</span><span class="o">.</span><span class="mf">0.1</span><span class="o">-</span><span class="mi">8080</span><span class="o">-</span><span class="mi">1</span><span class="o">)</span> <span class="n">rule</span><span class="o">.</span><span class="na">debug</span><span class="o">{</span><span class="n">Fail</span> <span class="mi">2</span><span class="n">PC</span> <span class="n">after</span> <span class="n">prepare</span><span class="o">}</span> <span class="o">:</span> <span class="o">!!!</span><span class="n">killing</span> <span class="n">JVM</span><span class="o">!!!</span>
</pre></div></li>
<li><p>NOTE: Until you restart JBoss server 1, you will see several error messages in JBoss server 2. These are to be expected:</p>
<div class="highlight"><pre><span class="m">15</span><span class="s-Atom">:</span><span class="m">46</span><span class="s-Atom">:</span><span class="m">55</span><span class="p">,</span><span class="m">044</span> <span class="nv">INFO</span>  <span class="p">[</span><span class="s-Atom">org</span><span class="p">.</span><span class="s-Atom">jboss</span><span class="p">.</span><span class="s-Atom">ejb</span><span class="p">.</span><span class="s-Atom">client</span><span class="p">]</span> <span class="p">(</span><span class="nv">RequestProcessor</span><span class="o">-</span><span class="m">10</span><span class="p">)</span> <span class="nv">JBoss</span> <span class="nv">EJB</span> <span class="nv">Client</span> <span class="s-Atom">version</span> <span class="m">1</span><span class="p">.</span><span class="m">0</span><span class="p">.</span><span class="m">0</span><span class="p">.</span><span class="nv">Beta12</span>
<span class="m">15</span><span class="s-Atom">:</span><span class="m">49</span><span class="s-Atom">:</span><span class="m">06</span><span class="p">,</span><span class="m">579</span> <span class="nv">WARN</span>  <span class="p">[</span><span class="s-Atom">com</span><span class="p">.</span><span class="s-Atom">arjuna</span><span class="p">.</span><span class="s-Atom">ats</span><span class="p">.</span><span class="s-Atom">jts</span><span class="p">]</span> <span class="p">(</span><span class="nv">Periodic</span> <span class="nv">Recovery</span><span class="p">)</span> <span class="nv">ARJUNA022167</span><span class="s-Atom">:</span> <span class="nv">Got</span> <span class="nv">TRANSIENT</span> <span class="s-Atom">from</span> <span class="nv">ORB</span> <span class="s-Atom">for</span> <span class="s-Atom">tx</span> <span class="m">0</span><span class="s-Atom">:</span><span class="nf">ffffc0a8013c</span><span class="o">:-</span><span class="m">2</span><span class="s-Atom">eb1158b:</span><span class="m">4</span><span class="s-Atom">f280ce3:</span><span class="m">1</span><span class="s-Atom">a</span><span class="p">,</span> <span class="s-Atom">unable</span> <span class="s-Atom">determine</span> <span class="s-Atom">status</span><span class="p">,</span> <span class="s-Atom">will</span> <span class="s-Atom">retry</span> <span class="s-Atom">later</span>
<span class="m">15</span><span class="s-Atom">:</span><span class="m">51</span><span class="s-Atom">:</span><span class="m">19</span><span class="p">,</span><span class="m">103</span> <span class="nv">WARN</span>  <span class="p">[</span><span class="s-Atom">com</span><span class="p">.</span><span class="s-Atom">arjuna</span><span class="p">.</span><span class="s-Atom">ats</span><span class="p">.</span><span class="s-Atom">jts</span><span class="p">]</span> <span class="p">(</span><span class="nv">Periodic</span> <span class="nv">Recovery</span><span class="p">)</span> <span class="nv">ARJUNA022167</span><span class="s-Atom">:</span> <span class="nv">Got</span> <span class="nv">TRANSIENT</span> <span class="s-Atom">from</span> <span class="nv">ORB</span> <span class="s-Atom">for</span> <span class="s-Atom">tx</span> <span class="m">0</span><span class="s-Atom">:</span><span class="nf">ffffc0a8013c</span><span class="o">:-</span><span class="m">2</span><span class="s-Atom">eb1158b:</span><span class="m">4</span><span class="s-Atom">f280ce3:</span><span class="m">1</span><span class="s-Atom">a</span><span class="p">,</span> <span class="s-Atom">unable</span> <span class="s-Atom">determine</span> <span class="s-Atom">status</span><span class="p">,</span> <span class="s-Atom">will</span> <span class="s-Atom">retry</span> <span class="s-Atom">later</span>
<span class="m">15</span><span class="s-Atom">:</span><span class="m">51</span><span class="s-Atom">:</span><span class="m">19</span><span class="p">,</span><span class="m">120</span> <span class="nv">WARN</span>  <span class="p">[</span><span class="s-Atom">com</span><span class="p">.</span><span class="s-Atom">arjuna</span><span class="p">.</span><span class="s-Atom">ats</span><span class="p">.</span><span class="s-Atom">jta</span><span class="p">]</span> <span class="p">(</span><span class="nv">Periodic</span> <span class="nv">Recovery</span><span class="p">)</span> <span class="nv">ARJUNA016005</span><span class="s-Atom">:</span> <span class="nv">JTS</span> <span class="nv">XARecoveryModule</span><span class="p">.</span><span class="s-Atom">xaRecovery</span> <span class="o">-</span> <span class="s-Atom">failed</span> <span class="s-Atom">to</span> <span class="s-Atom">recover</span> <span class="nv">XAResource</span><span class="p">.</span> <span class="s-Atom">status</span> <span class="o">is</span> <span class="err">$</span><span class="m">3</span>
<span class="m">15</span><span class="s-Atom">:</span><span class="m">53</span><span class="s-Atom">:</span><span class="m">31</span><span class="p">,</span><span class="m">638</span> <span class="nv">WARN</span>  <span class="p">[</span><span class="s-Atom">com</span><span class="p">.</span><span class="s-Atom">arjuna</span><span class="p">.</span><span class="s-Atom">ats</span><span class="p">.</span><span class="s-Atom">jts</span><span class="p">]</span> <span class="p">(</span><span class="nv">Periodic</span> <span class="nv">Recovery</span><span class="p">)</span> <span class="nv">ARJUNA022167</span><span class="s-Atom">:</span> <span class="nv">Got</span> <span class="nv">TRANSIENT</span> <span class="s-Atom">from</span> <span class="nv">ORB</span> <span class="s-Atom">for</span> <span class="s-Atom">tx</span> <span class="m">0</span><span class="s-Atom">:</span><span class="nf">ffffc0a8013c</span><span class="o">:-</span><span class="m">2</span><span class="s-Atom">eb1158b:</span><span class="m">4</span><span class="s-Atom">f280ce3:</span><span class="m">1</span><span class="s-Atom">a</span><span class="p">,</span> <span class="s-Atom">unable</span> <span class="s-Atom">determine</span> <span class="s-Atom">status</span><span class="p">,</span> <span class="s-Atom">will</span> <span class="s-Atom">retry</span> <span class="s-Atom">later</span>
<span class="m">15</span><span class="s-Atom">:</span><span class="m">53</span><span class="s-Atom">:</span><span class="m">31</span><span class="p">,</span><span class="m">644</span> <span class="nv">WARN</span>  <span class="p">[</span><span class="s-Atom">com</span><span class="p">.</span><span class="s-Atom">arjuna</span><span class="p">.</span><span class="s-Atom">ats</span><span class="p">.</span><span class="s-Atom">jta</span><span class="p">]</span> <span class="p">(</span><span class="nv">Periodic</span> <span class="nv">Recovery</span><span class="p">)</span> <span class="nv">ARJUNA016005</span><span class="s-Atom">:</span> <span class="nv">JTS</span> <span class="nv">XARecoveryModule</span><span class="p">.</span><span class="s-Atom">xaRecovery</span> <span class="o">-</span> <span class="s-Atom">failed</span> <span class="s-Atom">to</span> <span class="s-Atom">recover</span> <span class="nv">XAResource</span><span class="p">.</span> <span class="s-Atom">status</span> <span class="o">is</span> <span class="err">$</span><span class="m">3</span>
</pre></div></li>
</ul></li>
<li><p>At this point, Byteman halts or crashes server 1. You should also be able to view the contents of the object store by typing:</p>
<div class="highlight"><pre><span class="n">tree</span> <span class="n">server</span><span class="err">*/</span><span class="n">standalone</span><span class="o">/</span><span class="n">data</span><span class="o">/</span><span class="n">tx</span><span class="o">-</span><span class="n">object</span><span class="o">-</span><span class="n">store</span>
</pre></div>
<p>This should display:</p>
<div class="highlight"><pre><span class="n">server1</span><span class="o">/</span><span class="n">standalone</span><span class="o">/</span><span class="n">data</span><span class="o">/</span><span class="n">tx</span><span class="o">-</span><span class="n">object</span><span class="o">-</span><span class="n">store</span>
<span class="err">`</span><span class="o">--</span> <span class="n">ShadowNoFileLockStore</span>
    <span class="err">`</span><span class="o">--</span> <span class="n">defaultStore</span>
        <span class="o">|--</span> <span class="n">CosTransactions</span>
        <span class="o">|</span>   <span class="err">`</span><span class="o">--</span> <span class="n">XAResourceRecord</span>
        <span class="o">|</span>       <span class="err">`</span><span class="o">--</span> <span class="mi">0</span><span class="n">_ffffc0a8013c_38e104bd_4f280cdb_1d</span>
        <span class="o">|--</span> <span class="n">Recovery</span>
        <span class="o">|</span>   <span class="err">`</span><span class="o">--</span> <span class="n">FactoryContact</span>
        <span class="o">|</span>       <span class="o">|--</span> <span class="mi">0</span><span class="n">_ffffc0a8013c_38e104bd_4f280cdb_17</span>
        <span class="o">|</span>       <span class="o">|--</span> <span class="mi">0</span><span class="n">_ffffc0a8013c_</span><span class="o">-</span><span class="mi">671009</span><span class="n">a_4f280e7e_17</span>
        <span class="o">|</span>       <span class="err">`</span><span class="o">--</span> <span class="mi">0</span><span class="n">_ffffc0a8013c_6d5d82b5_4f280a16_f</span>
        <span class="o">|--</span> <span class="n">RecoveryCoordinator</span>
        <span class="o">|</span>   <span class="err">`</span><span class="o">--</span> <span class="mi">0</span><span class="n">_ffff52e38d0c_c91_4140398c_0</span>
        <span class="err">`</span><span class="o">--</span> <span class="n">StateManager</span>
            <span class="err">`</span><span class="o">--</span> <span class="n">BasicAction</span>
                <span class="err">`</span><span class="o">--</span> <span class="n">TwoPhaseCoordinator</span>
                    <span class="err">`</span><span class="o">--</span> <span class="n">ArjunaTransactionImple</span>
                        <span class="err">`</span><span class="o">--</span> <span class="mi">0</span><span class="n">_ffffc0a8013c_38e104bd_4f280cdb_19</span>
<span class="n">server2</span><span class="o">/</span><span class="n">standalone</span><span class="o">/</span><span class="n">data</span><span class="o">/</span><span class="n">tx</span><span class="o">-</span><span class="n">object</span><span class="o">-</span><span class="n">store</span>
<span class="err">`</span><span class="o">--</span> <span class="n">ShadowNoFileLockStore</span>
    <span class="err">`</span><span class="o">--</span> <span class="n">defaultStore</span>
        <span class="o">|--</span> <span class="n">CosTransactions</span>
        <span class="o">|</span>   <span class="err">`</span><span class="o">--</span> <span class="n">XAResourceRecord</span>
        <span class="o">|</span>       <span class="err">`</span><span class="o">--</span> <span class="mi">0</span><span class="n">_ffffc0a8013c_</span><span class="o">-</span><span class="mi">2</span><span class="n">eb1158b_4f280ce3_1e</span>
        <span class="o">|--</span> <span class="n">Recovery</span>
        <span class="o">|</span>   <span class="err">`</span><span class="o">--</span> <span class="n">FactoryContact</span>
        <span class="o">|</span>       <span class="o">|--</span> <span class="mi">0</span><span class="n">_ffffc0a8013c_</span><span class="o">-</span><span class="mi">2</span><span class="n">eb1158b_4f280ce3_18</span>
        <span class="o">|</span>       <span class="err">`</span><span class="o">--</span> <span class="mi">0</span><span class="n">_ffffc0a8013c_4f6459f0_4f280a24_f</span>
        <span class="o">|--</span> <span class="n">RecoveryCoordinator</span>
        <span class="o">|</span>   <span class="err">`</span><span class="o">--</span> <span class="mi">0</span><span class="n">_ffff52e38d0c_c91_4140398c_0</span>
        <span class="err">`</span><span class="o">--</span> <span class="n">StateManager</span>
            <span class="err">`</span><span class="o">--</span> <span class="n">BasicAction</span>
                <span class="err">`</span><span class="o">--</span> <span class="n">TwoPhaseCoordinator</span>
                    <span class="err">`</span><span class="o">--</span> <span class="n">ArjunaTransactionImple</span>
                        <span class="err">`</span><span class="o">--</span> <span class="n">ServerTransaction</span>
                            <span class="err">`</span><span class="o">--</span> <span class="mi">0</span><span class="n">_ffffc0a8013c_</span><span class="o">-</span><span class="mi">2</span><span class="n">eb1158b_4f280ce3_1a</span>
</pre></div></li>
<li><p><a href="../README.html#disable-the-byteman-script">Disable the Byteman script</a> by restoring the backup configuration file for server 1.</p></li>
<li><p>Follow the steps above to restart server 1 and wait for recovery to complete. </p>

<p><em>IMPORTANT: By default, the recovery process checks the transactional state every two minutes, therefore it can take a while for recovery to happen. Also recovery for each server will take place at its own recovery interval.</em></p>

<ul>
<li><p>You will know when recovery is complete for server 2 as you will see the following in application-server-2 console:</p>
<div class="highlight"><pre><span class="mi">12</span><span class="o">:</span><span class="mi">09</span><span class="o">:</span><span class="mi">38</span><span class="o">,</span><span class="mi">697</span> <span class="n">INFO</span>  <span class="o">[</span><span class="n">org</span><span class="o">.</span><span class="na">jboss</span><span class="o">.</span><span class="na">ejb</span><span class="o">.</span><span class="na">client</span><span class="o">]</span> <span class="o">(</span><span class="n">RequestProcessor</span><span class="o">-</span><span class="mi">10</span><span class="o">)</span> <span class="n">JBoss</span> <span class="n">EJB</span> <span class="n">Client</span> <span class="n">version</span> <span class="mf">1.0</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="na">Beta11</span>
<span class="mi">12</span><span class="o">:</span><span class="mi">09</span><span class="o">:</span><span class="mi">39</span><span class="o">,</span><span class="mi">204</span> <span class="n">INFO</span>  <span class="o">[</span><span class="kd">class</span> <span class="n">org</span><span class="o">.</span><span class="na">jboss</span><span class="o">.</span><span class="na">as</span><span class="o">.</span><span class="na">quickstarts</span><span class="o">.</span><span class="na">cmt</span><span class="o">.</span><span class="na">jts</span><span class="o">.</span><span class="na">mdb</span><span class="o">.</span><span class="na">HelloWorldMDB</span><span class="o">]</span> <span class="o">(</span><span class="n">Thread</span><span class="o">-</span><span class="mi">3</span> <span class="o">(</span><span class="n">group</span><span class="o">:</span><span class="n">HornetQ</span><span class="o">-</span><span class="n">client</span><span class="o">-</span><span class="n">global</span><span class="o">-</span><span class="n">threads</span><span class="o">-</span><span class="mi">649946595</span><span class="o">))</span> <span class="n">Received</span> <span class="n">Message</span><span class="o">:</span> <span class="n">Created</span> <span class="n">invoice</span> <span class="k">for</span> <span class="n">customer</span> <span class="n">named</span><span class="o">:</span> <span class="n">Tom</span>
</pre></div></li>
<li><p>NOTE: You will also get several stack traces in JBoss server 1 console during recovery, these are to be expected as not all resources are available at all stages of recovery.</p>
<div class="highlight"><pre><span class="mi">15</span><span class="o">:</span><span class="mi">55</span><span class="o">:</span><span class="mi">41</span><span class="o">,</span><span class="mi">706</span> <span class="n">WARN</span>  <span class="o">[</span><span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ats</span><span class="o">.</span><span class="na">jts</span><span class="o">]</span> <span class="o">(</span><span class="n">Thread</span><span class="o">-</span><span class="mi">84</span><span class="o">)</span> <span class="n">ARJUNA022223</span><span class="o">:</span> <span class="n">ExtendedResourceRecord</span><span class="o">.</span><span class="na">topLevelCommit</span> <span class="n">caught</span> <span class="n">exception</span><span class="o">:</span> <span class="n">org</span><span class="o">.</span><span class="na">omg</span><span class="o">.</span><span class="na">CORBA</span><span class="o">.</span><span class="na">OBJECT_NOT_EXIST</span><span class="o">:</span> <span class="n">Server</span><span class="o">-</span><span class="n">side</span> <span class="n">Exception</span><span class="o">:</span> <span class="n">unknown</span> <span class="n">oid</span>
    <span class="n">at</span> <span class="n">sun</span><span class="o">.</span><span class="na">reflect</span><span class="o">.</span><span class="na">NativeConstructorAccessorImpl</span><span class="o">.</span><span class="na">newInstance0</span><span class="o">(</span><span class="n">Native</span> <span class="n">Method</span><span class="o">)</span> <span class="o">[:</span><span class="mf">1.6</span><span class="o">.</span><span class="mi">0</span><span class="n">_22</span><span class="o">]</span>
    <span class="n">at</span> <span class="n">sun</span><span class="o">.</span><span class="na">reflect</span><span class="o">.</span><span class="na">NativeConstructorAccessorImpl</span><span class="o">.</span><span class="na">newInstance</span><span class="o">(</span><span class="n">NativeConstructorAccessorImpl</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">57</span><span class="o">)</span> <span class="o">[:</span><span class="mf">1.6</span><span class="o">.</span><span class="mi">0</span><span class="n">_22</span><span class="o">]</span>
    <span class="n">at</span> <span class="n">sun</span><span class="o">.</span><span class="na">reflect</span><span class="o">.</span><span class="na">DelegatingConstructorAccessorImpl</span><span class="o">.</span><span class="na">newInstance</span><span class="o">(</span><span class="n">DelegatingConstructorAccessorImpl</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">45</span><span class="o">)</span> <span class="o">[:</span><span class="mf">1.6</span><span class="o">.</span><span class="mi">0</span><span class="n">_22</span><span class="o">]</span>
    <span class="n">at</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">reflect</span><span class="o">.</span><span class="na">Constructor</span><span class="o">.</span><span class="na">newInstance</span><span class="o">(</span><span class="n">Constructor</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">532</span><span class="o">)</span> <span class="o">[:</span><span class="mf">1.6</span><span class="o">.</span><span class="mi">0</span><span class="n">_22</span><span class="o">]</span>
    <span class="n">at</span> <span class="n">org</span><span class="o">.</span><span class="na">jacorb</span><span class="o">.</span><span class="na">orb</span><span class="o">.</span><span class="na">SystemExceptionHelper</span><span class="o">.</span><span class="na">read</span><span class="o">(</span><span class="n">SystemExceptionHelper</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">223</span><span class="o">)</span> <span class="o">[</span><span class="n">jacorb</span><span class="o">-</span><span class="mf">2.3</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">jbossorg</span><span class="o">-</span><span class="mi">1</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">org</span><span class="o">.</span><span class="na">jacorb</span><span class="o">.</span><span class="na">orb</span><span class="o">.</span><span class="na">ReplyReceiver</span><span class="o">.</span><span class="na">getReply</span><span class="o">(</span><span class="n">ReplyReceiver</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">319</span><span class="o">)</span> <span class="o">[</span><span class="n">jacorb</span><span class="o">-</span><span class="mf">2.3</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">jbossorg</span><span class="o">-</span><span class="mi">1</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">org</span><span class="o">.</span><span class="na">jacorb</span><span class="o">.</span><span class="na">orb</span><span class="o">.</span><span class="na">Delegate</span><span class="o">.</span><span class="na">invoke_internal</span><span class="o">(</span><span class="n">Delegate</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">1090</span><span class="o">)</span> <span class="o">[</span><span class="n">jacorb</span><span class="o">-</span><span class="mf">2.3</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">jbossorg</span><span class="o">-</span><span class="mi">1</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">org</span><span class="o">.</span><span class="na">jacorb</span><span class="o">.</span><span class="na">orb</span><span class="o">.</span><span class="na">Delegate</span><span class="o">.</span><span class="na">invoke</span><span class="o">(</span><span class="n">Delegate</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">957</span><span class="o">)</span> <span class="o">[</span><span class="n">jacorb</span><span class="o">-</span><span class="mf">2.3</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">jbossorg</span><span class="o">-</span><span class="mi">1</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">org</span><span class="o">.</span><span class="na">omg</span><span class="o">.</span><span class="na">CORBA</span><span class="o">.</span><span class="na">portable</span><span class="o">.</span><span class="na">ObjectImpl</span><span class="o">.</span><span class="na">_invoke</span><span class="o">(</span><span class="n">ObjectImpl</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">80</span><span class="o">)</span> <span class="o">[</span><span class="n">jacorb</span><span class="o">-</span><span class="mf">2.3</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">jbossorg</span><span class="o">-</span><span class="mi">1</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ArjunaOTS</span><span class="o">.</span><span class="na">_ArjunaSubtranAwareResourceStub</span><span class="o">.</span><span class="na">commit</span><span class="o">(</span><span class="n">_ArjunaSubtranAwareResourceStub</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">252</span><span class="o">)</span> <span class="o">[</span><span class="n">jbossjts</span><span class="o">-</span><span class="mf">4.16</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">Final</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ats</span><span class="o">.</span><span class="na">internal</span><span class="o">.</span><span class="na">jts</span><span class="o">.</span><span class="na">resources</span><span class="o">.</span><span class="na">ExtendedResourceRecord</span><span class="o">.</span><span class="na">topLevelCommit</span><span class="o">(</span><span class="n">ExtendedResourceRecord</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">502</span><span class="o">)</span> <span class="o">[</span><span class="n">jbossjts</span><span class="o">-</span><span class="mf">4.16</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">Final</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ats</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">coordinator</span><span class="o">.</span><span class="na">BasicAction</span><span class="o">.</span><span class="na">doCommit</span><span class="o">(</span><span class="n">BasicAction</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">2753</span><span class="o">)</span> <span class="o">[</span><span class="n">jbossjts</span><span class="o">-</span><span class="mf">4.16</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">Final</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ats</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">coordinator</span><span class="o">.</span><span class="na">BasicAction</span><span class="o">.</span><span class="na">doCommit</span><span class="o">(</span><span class="n">BasicAction</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">2669</span><span class="o">)</span> <span class="o">[</span><span class="n">jbossjts</span><span class="o">-</span><span class="mf">4.16</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">Final</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ats</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">coordinator</span><span class="o">.</span><span class="na">BasicAction</span><span class="o">.</span><span class="na">phase2Commit</span><span class="o">(</span><span class="n">BasicAction</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">1804</span><span class="o">)</span> <span class="o">[</span><span class="n">jbossjts</span><span class="o">-</span><span class="mf">4.16</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">Final</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ats</span><span class="o">.</span><span class="na">internal</span><span class="o">.</span><span class="na">jts</span><span class="o">.</span><span class="na">recovery</span><span class="o">.</span><span class="na">transactions</span><span class="o">.</span><span class="na">RecoveredTransaction</span><span class="o">.</span><span class="na">replayPhase2</span><span class="o">(</span><span class="n">RecoveredTransaction</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">197</span><span class="o">)</span> <span class="o">[</span><span class="n">jbossjts</span><span class="o">-</span><span class="mf">4.16</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">Final</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ats</span><span class="o">.</span><span class="na">internal</span><span class="o">.</span><span class="na">jts</span><span class="o">.</span><span class="na">recovery</span><span class="o">.</span><span class="na">transactions</span><span class="o">.</span><span class="na">TransactionCache</span><span class="o">.</span><span class="na">replayPhase2</span><span class="o">(</span><span class="n">TransactionCache</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">233</span><span class="o">)</span> <span class="o">[</span><span class="n">jbossjts</span><span class="o">-</span><span class="mf">4.16</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">Final</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ats</span><span class="o">.</span><span class="na">internal</span><span class="o">.</span><span class="na">jts</span><span class="o">.</span><span class="na">recovery</span><span class="o">.</span><span class="na">transactions</span><span class="o">.</span><span class="na">CachedRecoveredTransaction</span><span class="o">.</span><span class="na">replayPhase2</span><span class="o">(</span><span class="n">CachedRecoveredTransaction</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">173</span><span class="o">)</span> <span class="o">[</span><span class="n">jbossjts</span><span class="o">-</span><span class="mf">4.16</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">Final</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ats</span><span class="o">.</span><span class="na">internal</span><span class="o">.</span><span class="na">jts</span><span class="o">.</span><span class="na">recovery</span><span class="o">.</span><span class="na">transactions</span><span class="o">.</span><span class="na">RecoveredTransactionReplayer</span><span class="o">.</span><span class="na">run</span><span class="o">(</span><span class="n">RecoveredTransactionReplayer</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">118</span><span class="o">)</span> <span class="o">[</span><span class="n">jbossjts</span><span class="o">-</span><span class="mf">4.16</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">Final</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>

<span class="mi">15</span><span class="o">:</span><span class="mi">55</span><span class="o">:</span><span class="mi">55</span><span class="o">,</span><span class="mi">179</span> <span class="n">WARN</span>  <span class="o">[</span><span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ats</span><span class="o">.</span><span class="na">jts</span><span class="o">]</span> <span class="o">(</span><span class="n">Periodic</span> <span class="n">Recovery</span><span class="o">)</span> <span class="n">ARJUNA022223</span><span class="o">:</span> <span class="n">ExtendedResourceRecord</span><span class="o">.</span><span class="na">topLevelCommit</span> <span class="n">caught</span> <span class="n">exception</span><span class="o">:</span> <span class="n">org</span><span class="o">.</span><span class="na">omg</span><span class="o">.</span><span class="na">CORBA</span><span class="o">.</span><span class="na">OBJECT_NOT_EXIST</span><span class="o">:</span> <span class="n">Server</span><span class="o">-</span><span class="n">side</span> <span class="n">Exception</span><span class="o">:</span> <span class="n">unknown</span> <span class="n">oid</span>
    <span class="n">at</span> <span class="n">sun</span><span class="o">.</span><span class="na">reflect</span><span class="o">.</span><span class="na">NativeConstructorAccessorImpl</span><span class="o">.</span><span class="na">newInstance0</span><span class="o">(</span><span class="n">Native</span> <span class="n">Method</span><span class="o">)</span> <span class="o">[:</span><span class="mf">1.6</span><span class="o">.</span><span class="mi">0</span><span class="n">_22</span><span class="o">]</span>
    <span class="n">at</span> <span class="n">sun</span><span class="o">.</span><span class="na">reflect</span><span class="o">.</span><span class="na">NativeConstructorAccessorImpl</span><span class="o">.</span><span class="na">newInstance</span><span class="o">(</span><span class="n">NativeConstructorAccessorImpl</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">57</span><span class="o">)</span> <span class="o">[:</span><span class="mf">1.6</span><span class="o">.</span><span class="mi">0</span><span class="n">_22</span><span class="o">]</span>
    <span class="n">at</span> <span class="n">sun</span><span class="o">.</span><span class="na">reflect</span><span class="o">.</span><span class="na">DelegatingConstructorAccessorImpl</span><span class="o">.</span><span class="na">newInstance</span><span class="o">(</span><span class="n">DelegatingConstructorAccessorImpl</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">45</span><span class="o">)</span> <span class="o">[:</span><span class="mf">1.6</span><span class="o">.</span><span class="mi">0</span><span class="n">_22</span><span class="o">]</span>
    <span class="n">at</span> <span class="n">java</span><span class="o">.</span><span class="na">lang</span><span class="o">.</span><span class="na">reflect</span><span class="o">.</span><span class="na">Constructor</span><span class="o">.</span><span class="na">newInstance</span><span class="o">(</span><span class="n">Constructor</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">532</span><span class="o">)</span> <span class="o">[:</span><span class="mf">1.6</span><span class="o">.</span><span class="mi">0</span><span class="n">_22</span><span class="o">]</span>
    <span class="n">at</span> <span class="n">org</span><span class="o">.</span><span class="na">jacorb</span><span class="o">.</span><span class="na">orb</span><span class="o">.</span><span class="na">SystemExceptionHelper</span><span class="o">.</span><span class="na">read</span><span class="o">(</span><span class="n">SystemExceptionHelper</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">223</span><span class="o">)</span> <span class="o">[</span><span class="n">jacorb</span><span class="o">-</span><span class="mf">2.3</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">jbossorg</span><span class="o">-</span><span class="mi">1</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">org</span><span class="o">.</span><span class="na">jacorb</span><span class="o">.</span><span class="na">orb</span><span class="o">.</span><span class="na">ReplyReceiver</span><span class="o">.</span><span class="na">getReply</span><span class="o">(</span><span class="n">ReplyReceiver</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">319</span><span class="o">)</span> <span class="o">[</span><span class="n">jacorb</span><span class="o">-</span><span class="mf">2.3</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">jbossorg</span><span class="o">-</span><span class="mi">1</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">org</span><span class="o">.</span><span class="na">jacorb</span><span class="o">.</span><span class="na">orb</span><span class="o">.</span><span class="na">Delegate</span><span class="o">.</span><span class="na">invoke_internal</span><span class="o">(</span><span class="n">Delegate</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">1090</span><span class="o">)</span> <span class="o">[</span><span class="n">jacorb</span><span class="o">-</span><span class="mf">2.3</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">jbossorg</span><span class="o">-</span><span class="mi">1</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">org</span><span class="o">.</span><span class="na">jacorb</span><span class="o">.</span><span class="na">orb</span><span class="o">.</span><span class="na">Delegate</span><span class="o">.</span><span class="na">invoke</span><span class="o">(</span><span class="n">Delegate</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">957</span><span class="o">)</span> <span class="o">[</span><span class="n">jacorb</span><span class="o">-</span><span class="mf">2.3</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">jbossorg</span><span class="o">-</span><span class="mi">1</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">org</span><span class="o">.</span><span class="na">omg</span><span class="o">.</span><span class="na">CORBA</span><span class="o">.</span><span class="na">portable</span><span class="o">.</span><span class="na">ObjectImpl</span><span class="o">.</span><span class="na">_invoke</span><span class="o">(</span><span class="n">ObjectImpl</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">80</span><span class="o">)</span> <span class="o">[</span><span class="n">jacorb</span><span class="o">-</span><span class="mf">2.3</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">jbossorg</span><span class="o">-</span><span class="mi">1</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ArjunaOTS</span><span class="o">.</span><span class="na">_ArjunaSubtranAwareResourceStub</span><span class="o">.</span><span class="na">commit</span><span class="o">(</span><span class="n">_ArjunaSubtranAwareResourceStub</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">252</span><span class="o">)</span> <span class="o">[</span><span class="n">jbossjts</span><span class="o">-</span><span class="mf">4.16</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">Final</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ats</span><span class="o">.</span><span class="na">internal</span><span class="o">.</span><span class="na">jts</span><span class="o">.</span><span class="na">resources</span><span class="o">.</span><span class="na">ExtendedResourceRecord</span><span class="o">.</span><span class="na">topLevelCommit</span><span class="o">(</span><span class="n">ExtendedResourceRecord</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">502</span><span class="o">)</span> <span class="o">[</span><span class="n">jbossjts</span><span class="o">-</span><span class="mf">4.16</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">Final</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ats</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">coordinator</span><span class="o">.</span><span class="na">BasicAction</span><span class="o">.</span><span class="na">doCommit</span><span class="o">(</span><span class="n">BasicAction</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">2753</span><span class="o">)</span> <span class="o">[</span><span class="n">jbossjts</span><span class="o">-</span><span class="mf">4.16</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">Final</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ats</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">coordinator</span><span class="o">.</span><span class="na">BasicAction</span><span class="o">.</span><span class="na">doCommit</span><span class="o">(</span><span class="n">BasicAction</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">2669</span><span class="o">)</span> <span class="o">[</span><span class="n">jbossjts</span><span class="o">-</span><span class="mf">4.16</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">Final</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ats</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">coordinator</span><span class="o">.</span><span class="na">BasicAction</span><span class="o">.</span><span class="na">phase2Commit</span><span class="o">(</span><span class="n">BasicAction</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">1804</span><span class="o">)</span> <span class="o">[</span><span class="n">jbossjts</span><span class="o">-</span><span class="mf">4.16</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">Final</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ats</span><span class="o">.</span><span class="na">internal</span><span class="o">.</span><span class="na">jts</span><span class="o">.</span><span class="na">recovery</span><span class="o">.</span><span class="na">transactions</span><span class="o">.</span><span class="na">RecoveredTransaction</span><span class="o">.</span><span class="na">replayPhase2</span><span class="o">(</span><span class="n">RecoveredTransaction</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">197</span><span class="o">)</span> <span class="o">[</span><span class="n">jbossjts</span><span class="o">-</span><span class="mf">4.16</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">Final</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ats</span><span class="o">.</span><span class="na">internal</span><span class="o">.</span><span class="na">jts</span><span class="o">.</span><span class="na">recovery</span><span class="o">.</span><span class="na">transactions</span><span class="o">.</span><span class="na">TransactionCache</span><span class="o">.</span><span class="na">replayPhase2</span><span class="o">(</span><span class="n">TransactionCache</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">233</span><span class="o">)</span> <span class="o">[</span><span class="n">jbossjts</span><span class="o">-</span><span class="mf">4.16</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">Final</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ats</span><span class="o">.</span><span class="na">internal</span><span class="o">.</span><span class="na">jts</span><span class="o">.</span><span class="na">recovery</span><span class="o">.</span><span class="na">transactions</span><span class="o">.</span><span class="na">CachedRecoveredTransaction</span><span class="o">.</span><span class="na">replayPhase2</span><span class="o">(</span><span class="n">CachedRecoveredTransaction</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">173</span><span class="o">)</span> <span class="o">[</span><span class="n">jbossjts</span><span class="o">-</span><span class="mf">4.16</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">Final</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ats</span><span class="o">.</span><span class="na">internal</span><span class="o">.</span><span class="na">jts</span><span class="o">.</span><span class="na">recovery</span><span class="o">.</span><span class="na">transactions</span><span class="o">.</span><span class="na">TransactionRecoveryModule</span><span class="o">.</span><span class="na">recoverTransaction</span><span class="o">(</span><span class="n">TransactionRecoveryModule</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">217</span><span class="o">)</span> <span class="o">[</span><span class="n">jbossjts</span><span class="o">-</span><span class="mf">4.16</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">Final</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ats</span><span class="o">.</span><span class="na">internal</span><span class="o">.</span><span class="na">jts</span><span class="o">.</span><span class="na">recovery</span><span class="o">.</span><span class="na">transactions</span><span class="o">.</span><span class="na">TransactionRecoveryModule</span><span class="o">.</span><span class="na">periodicWorkSecondPass</span><span class="o">(</span><span class="n">TransactionRecoveryModule</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">161</span><span class="o">)</span> <span class="o">[</span><span class="n">jbossjts</span><span class="o">-</span><span class="mf">4.16</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">Final</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ats</span><span class="o">.</span><span class="na">internal</span><span class="o">.</span><span class="na">jts</span><span class="o">.</span><span class="na">recovery</span><span class="o">.</span><span class="na">transactions</span><span class="o">.</span><span class="na">TopLevelTransactionRecoveryModule</span><span class="o">.</span><span class="na">periodicWorkSecondPass</span><span class="o">(</span><span class="n">TopLevelTransactionRecoveryModule</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">81</span><span class="o">)</span> <span class="o">[</span><span class="n">jbossjts</span><span class="o">-</span><span class="mf">4.16</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">Final</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ats</span><span class="o">.</span><span class="na">internal</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">recovery</span><span class="o">.</span><span class="na">PeriodicRecovery</span><span class="o">.</span><span class="na">doWorkInternal</span><span class="o">(</span><span class="n">PeriodicRecovery</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">789</span><span class="o">)</span> <span class="o">[</span><span class="n">jbossjts</span><span class="o">-</span><span class="mf">4.16</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">Final</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
    <span class="n">at</span> <span class="n">com</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">ats</span><span class="o">.</span><span class="na">internal</span><span class="o">.</span><span class="na">arjuna</span><span class="o">.</span><span class="na">recovery</span><span class="o">.</span><span class="na">PeriodicRecovery</span><span class="o">.</span><span class="na">run</span><span class="o">(</span><span class="n">PeriodicRecovery</span><span class="o">.</span><span class="na">java</span><span class="o">:</span><span class="mi">371</span><span class="o">)</span> <span class="o">[</span><span class="n">jbossjts</span><span class="o">-</span><span class="mf">4.16</span><span class="o">.</span><span class="mi">1</span><span class="o">.</span><span class="na">Final</span><span class="o">.</span><span class="na">jar</span><span class="o">:]</span>
</pre></div></li>
<li><p>The easiest way to check when JBoss server 1 is recovered is to look in the object store and check that all the records are now cleaned up. The records that should be cleared are the ones in the defaultStore/CosTransactions/XAResourceRecord and defaultStore/StateManager/BasicAction/TwoPhaseCoordinator/ArjunaTransactionImple. </p></li>
<li><p>Records will remain in defaultStore/Recovery/FactoryContact and defaultStore/RecoveryCoordinator and that is to be expected. Run:</p>
<div class="highlight"><pre><span class="n">tree</span> <span class="n">server</span><span class="err">*/</span><span class="n">standalone</span><span class="o">/</span><span class="n">data</span><span class="o">/</span><span class="n">tx</span><span class="o">-</span><span class="n">object</span><span class="o">-</span><span class="n">store</span>
</pre></div></li>
<li><p>You should see this output:</p>
<div class="highlight"><pre><span class="n">server1</span><span class="o">/</span><span class="n">standalone</span><span class="o">/</span><span class="n">data</span><span class="o">/</span><span class="n">tx</span><span class="o">-</span><span class="n">object</span><span class="o">-</span><span class="n">store</span>
<span class="err">`</span><span class="o">--</span> <span class="n">ShadowNoFileLockStore</span>
    <span class="err">`</span><span class="o">--</span> <span class="n">defaultStore</span>
        <span class="o">|--</span> <span class="n">CosTransactions</span>
        <span class="o">|</span>   <span class="err">`</span><span class="o">--</span> <span class="n">XAResourceRecord</span>
        <span class="o">|--</span> <span class="n">Recovery</span>
        <span class="o">|</span>   <span class="err">`</span><span class="o">--</span> <span class="n">FactoryContact</span>
        <span class="o">|</span>       <span class="o">|--</span> <span class="mi">0</span><span class="n">_ffffc0a8013c_38e104bd_4f280cdb_17</span>
        <span class="o">|</span>       <span class="o">|--</span> <span class="mi">0</span><span class="n">_ffffc0a8013c_</span><span class="o">-</span><span class="mi">671009</span><span class="n">a_4f280e7e_17</span>
        <span class="o">|</span>       <span class="err">`</span><span class="o">--</span> <span class="mi">0</span><span class="n">_ffffc0a8013c_6d5d82b5_4f280a16_f</span>
        <span class="o">|--</span> <span class="n">RecoveryCoordinator</span>
        <span class="o">|</span>   <span class="err">`</span><span class="o">--</span> <span class="mi">0</span><span class="n">_ffff52e38d0c_c91_4140398c_0</span>
        <span class="err">`</span><span class="o">--</span> <span class="n">StateManager</span>
            <span class="err">`</span><span class="o">--</span> <span class="n">BasicAction</span>
                <span class="err">`</span><span class="o">--</span> <span class="n">TwoPhaseCoordinator</span>
                  <span class="err">`</span><span class="o">--</span> <span class="n">ArjunaTransactionImple</span>
<span class="n">server2</span><span class="o">/</span><span class="n">standalone</span><span class="o">/</span><span class="n">data</span><span class="o">/</span><span class="n">tx</span><span class="o">-</span><span class="n">object</span><span class="o">-</span><span class="n">store</span>
<span class="err">`</span><span class="o">--</span> <span class="n">ShadowNoFileLockStore</span>
    <span class="err">`</span><span class="o">--</span> <span class="n">defaultStore</span>
        <span class="o">|--</span> <span class="n">CosTransactions</span>
        <span class="o">|</span>   <span class="err">`</span><span class="o">--</span> <span class="n">XAResourceRecord</span>
        <span class="o">|--</span> <span class="n">Recovery</span>
        <span class="o">|</span>   <span class="err">`</span><span class="o">--</span> <span class="n">FactoryContact</span>
        <span class="o">|</span>       <span class="o">|--</span> <span class="mi">0</span><span class="n">_ffffc0a8013c_</span><span class="o">-</span><span class="mi">2</span><span class="n">eb1158b_4f280ce3_18</span>
        <span class="o">|</span>       <span class="err">`</span><span class="o">--</span> <span class="mi">0</span><span class="n">_ffffc0a8013c_4f6459f0_4f280a24_f</span>
        <span class="o">|--</span> <span class="n">RecoveryCoordinator</span>
        <span class="o">|</span>   <span class="err">`</span><span class="o">--</span> <span class="mi">0</span><span class="n">_ffff52e38d0c_c91_4140398c_0</span>
        <span class="err">`</span><span class="o">--</span> <span class="n">StateManager</span>
            <span class="err">`</span><span class="o">--</span> <span class="n">BasicAction</span>
                <span class="err">`</span><span class="o">--</span> <span class="n">TwoPhaseCoordinator</span>
                    <span class="err">`</span><span class="o">--</span> <span class="n">ArjunaTransactionImple</span>
                        <span class="err">`</span><span class="o">--</span> <span class="n">ServerTransaction</span>
</pre></div></li>
</ul></li>
<li><p>After recovery is complete, access the application URL <a href="http://localhost:8080/jboss-jts-application-component-1/customers.jsf">http://localhost:8080/jboss-jts-application-component-1/customers.jsf</a>. The user you created should now appear in the list.</p></li>
<li><p>Do NOT forget to <a href="../README.html#disable-the-byteman-script">disable the Byteman script</a> by restoring the backup server configuration file. The Byteman rule must be removed to ensure that your application server will be able to commit 2PC transactions!</p></li>
</ol>
</body></html>
